Automated data transfer from mobile application silos to authorized third-party applications

ABSTRACT

A “Data Transfer Tool” extracts, labels and stores user data or information that may be confined within application silos during user interaction with arbitrary apps. The Data Transfer Tool enables sharing of this otherwise siloed data shared across and between authorized apps. The Data Transfer Tool provides a task-centric approach to define and extract structured and semantically meaningful information from source applications by providing multiple semantic models that are individually tailored to particular source apps. The Data Transfer Tool applies an accessibility API or the like of the OS in combination with the semantic model for the source app to scrape user entered or selected data or information from the source app. The Data Transfer Tool enables and creates new user experiences and increases user efficiency when interacting with various apps by making the scraped data or information available to subscribing destination apps approved to receive that data or information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation of U.S. patent application Ser. No.14/852,451, filed Sep. 11, 2015, which claims priority to IndiaProvisional Patent Application Serial Number 3847/CHE/2015, filed onJul. 27, 2015 and entitled “AUTOMATED DATA TRANSFER FROM MOBILEAPPLICATION SILOS TO AUTHORIZED THIRD-PARTY APPLICATIONS,” the entirecontents of each of which are hereby incorporated herein by referencefor all purposes.

BACKGROUND

Applications that make use of user data entry or selection typicallysilo the user-entered data. This includes data generated bothexplicitly, such as the user actually booking a flight through an app,as well as implicitly, such as browsing reviews of restaurants through areview app. Consequently, user entered data is not generally availableto other unrelated applications.

SUMMARY

The following Summary is provided to introduce a selection of conceptsin a simplified form that are further described below in the DetailedDescription. This Summary is not intended to identify key features oressential features of the claimed subject matter, nor is it intended tobe used as an aid in determining the scope of the claimed subjectmatter. Further, while certain disadvantages of other technologies maybe noted or discussed herein, the claimed subject matter is not intendedto be limited to implementations that may solve or address any or all ofthe disadvantages of those other technologies. The sole purpose of thisSummary is to present some concepts of the claimed subject matter in asimplified form as a prelude to the more detailed description that ispresented below.

In general, a “Data Transfer Tool,” as described herein, providesvarious techniques for extracting, labeling and storing user data thatis otherwise confined within application silos during user data entry.Consequently, the Data Transfer Tool enables otherwise siloed data to beshared across and between a plurality of authorized applications (alsoreferred to as “apps”) for the benefit of the user. In variousimplementations, the Data Transfer Tool applies an accessibilityapplication programming interface (API), or an assist API or the like toextract structured user data from a presentation layer of the operatingsystem (OS) in response to user interaction with arbitrary sourceapplications, without any modifications to the arbitrary sourceapplication code or binary. In other words, the Data Transfer Toolapplies any API that may be used to obtain data directly from thepresentation layer, regardless of the intended purpose of the API. TheData Transfer Tool then provides various mechanisms for sharing theextracted data to be shared with other authorized destinationapplications.

In various implementations, the extracted data is formatted as anapp-independent task. In general, a task represents a particularactivity and associated attributes or properties. So, a task can beconsidered as a template with variables (attributes or properties) thatare filled to complete the task template. For example, a flight-bookingtask can include attributes such as date, departure city, destinationcity, airline, etc.

More specifically, in various implementations, the Data Transfer Toolprovides a task-centric approach to define and extract structured andsemantically meaningful information from source applications as usersinteract with those applications. In various implementations, the DataTransfer Tool provides a plurality of semantic models that are eachindividually tailored to a particular source application. Then, when auser interacts with a particular source application, the Data TransferTool applies an accessibility API, or an assist API or the like incombination with the semantic model for the particular sourceapplication to scrape or otherwise extract user entered or selected dataor information from the presentation layer of the OS. The Data TransferTool then enables and creates new user experiences and increases userefficiency when interacting with destination applications by making thescraped data or information available to one or more subscribingdestination applications that are approved to receive that data orinformation.

For example, in various implementations, the Data Transfer Tool appliesan API such as an accessibility API, or assist API or the like toextract a plurality of data items from text being rendered on a displaydevice by a presentation subsystem or layer of the OS in response touser interaction with a particular source application. The Data TransferTool then applies a predefined application content model correspondingto that source application to identify and label the extracted dataitems. In addition, in various implementations, a subscription processor the like is applied to authorize (e.g., set an authorization state)and subscribe one or more third-party applications to access theidentified and labeled extracted data items. Finally, in variousimplementations, the Data Transfer Tool automatically applies one ormore of the identified and labeled extracted data items to populate oneor more data fields of one or more of the authorized third-partydestination applications.

The Data Transfer Tool described herein provides various techniques forextracting, labeling and storing user data siloed by a sourceapplication so that the stored data can be shared across and between aplurality of authorized destination applications to improve userinteraction efficiency with the authorized applications. In addition tothe benefits described above, other advantages of the Data Transfer Toolwill become apparent from the detailed description that followshereinafter.

BRIEF DESCRIPTION OF THE DRAWINGS

The specific features, aspects, and advantages of the claimed subjectmatter will become better understood with regard to the followingdescription, appended claims, and accompanying drawings where:

FIG. 1 provides an exemplary architectural flow diagram that illustratesprogram modules for effecting various implementations for training appmodels for use by a “Data Transfer Tool,” as described herein.

FIG. 2 provides an exemplary architectural flow diagram that illustratesprogram modules for effecting various implementations of the DataTransfer Tool for sharing data with authorized third-party applications,as described herein.

FIG. 3 provides a general system flow diagram that illustrates anexemplary implementation of the Data Transfer Tool, as described herein.

FIG. 4 provides a general system flow diagram that illustrates anexemplary implementation of the Data Transfer Tool, as described herein.

FIG. 5 provides a general system flow diagram that illustrates anexemplary implementation of an in-app assistant app of the Data TransferTool, as described herein.

FIG. 6 provides a general system diagram depicting a simplifiedgeneral-purpose computing device having simplified computing and I/Ocapabilities for use in effecting various implementations of the DataTransfer Tool, as described herein.

DETAILED DESCRIPTION

In the following description of various implementations of a “DataTransfer Tool,” reference is made to the accompanying drawings, whichform a part hereof, and in which is shown by way of illustrationspecific implementations in which the Data Transfer Tool may bepracticed. It should be understood that other implementations may beutilized and structural changes may be made without departing from thescope thereof.

Specific terminology will be resorted to in describing the variousimplementations described herein, and that it is not intended for theseimplementations to be limited to the specific terms so chosen.Furthermore, each specific term includes all its technical equivalentsthat operate in a broadly similar manner to achieve a similar purpose.Reference herein to “one implementation,” or “another implementation,”or an “exemplary implementation,” or an “alternate implementation” orsimilar phrases, means that a particular feature, a particularstructure, or particular characteristics described in connection withthe implementation can be included in at least one implementation of theData Transfer Tool. Further, the appearance of such phrases throughoutthe specification are not necessarily all referring to the sameimplementation, and separate or alternative implementations are notmutually exclusive of other implementations. The order described orillustrated herein for any process flows representing one or moreimplementations of the Data Transfer Tool does not inherently indicateany requirement for the processes to be implemented in the orderdescribed or illustrated, and any such order described or illustratedherein for any process flows do not imply any limitations of the DataTransfer Tool.

As utilized herein, the terms “component,” “system,” “client” and thelike are intended to refer to a computer-related entity, eitherhardware, software (e.g., in execution), firmware, or a combinationthereof. For example, a component can be a process running on aprocessor, an object, an executable, a program, a function, a library, asubroutine, a computer, or a combination of software and hardware. Byway of illustration, both an application running on a server and theserver can be a component. One or more components can reside within aprocess and a component can be localized on one computer and/ordistributed between two or more computers. The term “processor” isgenerally understood to refer to a hardware component, such as aprocessing unit of a computer system.

Furthermore, to the extent that the terms “includes,” “including,”“has,” “contains,” variants thereof, and other similar words are used ineither this detailed description or the claims, these terms are intendedto be inclusive in a manner similar to the term “comprising” as an opentransition word without precluding any additional or other elements.

1.0 Introduction:

Applications (also referred to herein as “apps”) are used to perform awide variety of operations, e.g., communicate with other users, populatea user calendar, search for online content, call a cab, order a pizza,book a hotel room, purchase items, download or stream music, movies orother content, read or enter reviews, etc. Such operations inherentlymake use of structured information, including, but not limited to,message content, user name, pick-up and drop-off locations, currentaddress, arrival date and time, search terms, item selection, etc. Userdata entry and/or selections when performing such operations aretypically siloed or otherwise held private by the app used to performthose operations.

In general, a “Data Transfer Tool,” as described herein, providesvarious techniques for extracting, labeling and storing user dataresulting from interaction with arbitrary apps during user data entry orselection. The Data Transfer Tool automatically formats this informationas an app-independent “task” that is no longer dependent on the app fromwhich it was extracted. The Data Transfer Tool then enables the tasksrepresenting this otherwise siloed or private data to be shared acrossand between authorized apps. Apps from which data is extracted arereferred to herein as “source apps.” Conversely, apps subscribing to orotherwise consuming or using the data extracted from source apps arereferred to herein as “destination apps.” In various implementations,any particular app may act as either or both a source application and adestination application. As such, any particular app may be both thesource of user entered or selected data or information, and a consumerof user entered or selected data or information extracted from otherapps.

Authorized destination apps may or may not have any relation orinteraction capabilities with the arbitrary source app from which thetask was extracted. Consequently, the Data Transfer Tool enables datasharing between source and destination apps that may be unrelated andotherwise incompatible, thereby enhancing and improving interactivitybetween arbitrary apps while increasing user efficiency when interactingwith those apps. Further, various data sharing features of the DataTransfer Tool enable and create a wide range of new user experienceswhile improving and increasing user efficiency when interacting withapps by sharing data or information with subscribing destination appsthat are approved to receive that data or information from one or moresource apps.

In various implementations, the Data Transfer Tool provides a userinterface (UI) or the like to specify whether any particular app isallowed to act as a source app. In other words, the user may choose toeither allow or prevent particular apps from acting as a source of datato any other apps. Further, in various implementations, subscriptions totasks by one or more destination apps may be authorized (or cancelled)via the user interface on either a global basis (e.g., allowed to accessall tasks, or a particular specified subset of tasks, from all sourceapps), or on a per-source basis (e.g., allowed to access all tasks, or aspecified subset of tasks, from one or more specified source apps). Inother words, the user is provided with a UI or the like to setauthorization states (e.g., authorized or denied) in response to tasksubscription requests.

In various implementations, the Data Transfer Tool includes anindication of the original source app from which the task was extracted.Additional metadata relating to the task creation, such as date, time,user location, etc., may optionally be included with the task forconsumption by authorized destination apps. In various implementations,the UI of the Data Transfer Tool may be applied to separately authorize(or deny) particular destination apps to consume metadata associatedwith otherwise authorized tasks.

In various implementations, data extraction is accomplished by applyingan accessibility application programming interface (API), or an assistAPI or the like to a presentation layer of the operating system (OS) orsource app to extract textual information from content being rendered orotherwise presented by the source app. In other words, the Data TransferTool applies any API that may be used to obtain data directly from thepresentation layer, regardless of the intended purpose of the API. Inparticular, API's such as accessibility or assist API's and the like areadapted and applied to extract or scrape user entered or selected dataor information from content and/or controls (e.g., radio buttons, checkboxes, drop-down lists, etc.) being rendered on a display by anarbitrary source app as the user interacts with that app. For purposesof discussion, the user entered or selected data or informationextracted from the app may be referred to herein as “extracted content”or similar terms.

Once the API has been applied to obtain extracted content, the DataTransfer Tool then applies a corresponding app model to the extractedcontent to automatically label and format the extracted content asstructured information represented by app-independent tasks. In general,each app model specifies a list of UI elements of the source app thatare to be tracked and logged. User input values or selections (e.g.,attributes) for these UI elements are extracted from the logs, and thenformatted as tasks that are stored or otherwise logged and published toauthorized subscribing apps.

Since tasks are independent of the app from which they were generated,using two different source apps to perform the same operations, e.g.,booking identical flights for the same user, may result in two identicaltasks (which may have different metadata) regardless of how theunderlying information was originally selected or presented by eithersource app. In this sense, a task is defined herein as an action (e.g.,booked flight, viewed or entered review, placed item in digital shoppingcart, etc.) in combination with one or more action-specific attributes(e.g., origin, destination, time, review content, item identification,etc.).

For example, in various implementations, when the user books a flight inany flight app (or performs any data selection or entry in any othersource app), the Data Transfer Tool may generate a task that includesattributes of arrival time and arrival airport relating to the flightbooking action. In various implementations, the user may be notified ofthe option to share the resulting flight booking task (or task relatingto some other data entry or selection) with one or more destination appsthat have subscribed to or requested this task. For example, anauthorized cab app may subscribe to the flight-booking task (withoutcaring where or how that task originates). This would allow the user toeasily book a cab using the authorized cab app for a time shortly afterher flight is scheduled to arrive without needing to reenter informationsuch as name, arrival airport, arrival time, etc.

Further, in various implementations, the user may be notified when atask from a source app is being provided to a destination app, unlessprior authorization has been granted, in which case the correspondingtask may be provided to the authorized destination app withoutnotification to the user. If the subscribing destination app (e.g., theaforementioned cab app) is authorized to share the user's flight bookingdata (or other task), the Data Transfer Tool automatically passes therequested task to the authorized app for use by the authorized app toautomatically begin or complete some operation (e.g., call a cab) or topopulate some data field or make a selection in a UI widget.Consequently, the data transfer features of the Data Transfer Toolimprove user efficiency by eliminating some or all of the manual dataentry that would otherwise be requested by the authorized app.

1.1 System Overview:

As noted above, the “Data Transfer Tool,” provides various techniquesfor extracting, labeling and storing user data siloed by a sourceapplication so that the stored data can be shared across and between aplurality of authorized destination applications. The processessummarized above are illustrated by the general system diagrams of FIG.1 and FIG. 2. In particular, the system diagram of FIG. 1 illustratesthe interrelationships between program modules for implementing variousimplementations of the Data Transfer Tool for training app models, asdescribed herein. In addition, the system diagram of FIG. 2 illustratesthe interrelationships between program modules for implementing variousimplementations of the Data Transfer Tool for sharing data withauthorized third-party applications, as described herein.

Furthermore, while the system diagram of FIG. 1 and FIG. 2 illustratehigh-level views of various implementations of the Data Transfer Tool,these figures are not intended to provide an exhaustive or completeillustration of every possible implementation of the Data Transfer Toolas described throughout this document.

In addition, any boxes and interconnections between boxes that may berepresented by broken or dashed lines in either or both FIG. 1 and FIG.2 represent alternate implementations of the Data Transfer Tooldescribed herein, and any or all of these alternate implementations, asdescribed below, may be used in combination with other alternateimplementations that are described throughout this document.

In general, as illustrated by FIG. 1, in various implementations, theapp model training processes enabled by the Data Transfer Tool beginsoperation with a UI Scraper Module 100 that applies OS instrumentation(not shown) or an API Module 105 (e.g., an Accessibility API, an“Assist” API, etc.) to extract content from an app presentation layer110 of arbitrary source apps 115 during user interaction with thatsource app. The extracted content (e.g., scraped content 130) is thenlogged or stored to a database, library or other storage for subsequentuse by the Data Transfer Tool.

Both the OS instrumentation or API Module 105 generally operate byextracting content from the app presentation layer 110 of the arbitrarysource apps 115 during user interaction with those arbitrary sourceapps. In general, a UI Module 120 provides a user interface that enablesthe user to interact with source apps 115 via typical input methods,including, but not limited to, keyboards, pointing devices, touch,gestures, speech, etc.).

In various implementations, during the app model training process, anoptional Directed Interaction Module 125 directs one or more users toperform a sequence of one or more predefined primary and secondaryactions via the UI of particular source apps 115. The scraped content130 is captured during this directed user interaction to provide abaseline that enables the Data Transfer Tool to infer a mapping betweenuser actions and UI widgets to particular user entered data orselections. See Section 2.3.1 for a discussion of app model constructionvia guided human interaction.

In various implementations, an Inference Engine Module 135 is thenapplied by the Data Transfer Tool to process the scraped content 130 toinfer an app model 140 for each particular source app 115. In variousimplementations, an App Model Update Module 145 is applied by the DataTransfer Tool to automatically update one or more of the app models 140in response to changes or updates to the corresponding source app 115.

As illustrated by FIG. 2, once one or more app models 140 have beeninferred or otherwise constructed by the Data Transfer Tool, the DataTransfer is capable of sharing data scraped or otherwise extracted fromarbitrary source apps with one or more authorized destination apps. Forexample, as illustrated in FIG. 2, the UI Scraper Module 100 againapplies OS instrumentation (not shown) or the API Module 105 to extractcontent from an app presentation layer 110 of arbitrary source apps 115during user interaction with that source app via the UI module 120. Theresulting scraped content 130 is again logged or stored to a database,library or other storage for subsequent use by the Data Transfer Tool.

However, in contrast to the use of the scrapped content 130 shown inFIG. 1, because the app models 140 have already been created, theprocesses illustrated by FIG. 2 continues with a Task Output Module 200that matches app models 140 to the arbitrary source applications fromwhich particular elements of the scraped content 130 were generated, andthen applies the appropriate app model to populate one or more tasktemplates using the scraped content extracted from the source app inresponse to user interaction with the source app. The populated tasktemplates are then stored to a library or database of tasks 210.

A Task Application Module 220 then automatically populates one or moredata fields and/or U I widgets of authorized destination apps 230 basedon subscriptions to those tasks by one or more authorized destinationapps. Authorizations for these subscriptions are maintained by aSubscription Module 240. In general, the subscription module 240provides a user interface that enables the user to grant, deny or changetask subscription authorizations requested by one or more destinationapps.

2.0 Operational Details of the Data Transfer Tool:

The above-described program modules are employed for implementingvarious implementations of the Data Transfer Tool. As summarized above,the Data Transfer Tool provides various techniques for extracting,labeling and storing user data siloed or otherwise held private by asource application so that the stored data can be shared across andbetween a plurality of authorized destination applications. Thefollowing sections provide a detailed discussion of the operation ofvarious implementations of the Data Transfer Tool, and of exemplarymethods for implementing the program modules described in Section 1 withrespect to FIG. 1. In particular, the following sections providesexamples and operational details of various implementations of the DataTransfer Tool, including:

-   -   Operational Overview of the Data Transfer Tool;    -   Information Extraction from Arbitrary Source Apps;    -   App-Specific Semantic Models of Source Apps;    -   Exemplary Cross-App Data Sharing Apps; and    -   Additional Implementations of the Data Transfer Tool.

2.1 Operational Overview:

Much of what is of interest to users and about users is contained withinthe apps operated by users. For example, while performing a search, amobile user might be as interested in finding a relevant app page (e.g.,a comment they had read earlier on some web page) as in searchingthrough their emails. As another example, an app session that results inan incomplete transaction (e.g., a search for a flight that stops shortof an actual purchase) might not generate a confirmation email or otherexternal indication but could still yield a useful signal or taskrelating to the user's intent. Such signals can be used directly for theuser's benefit by a variety of third-party apps, personal digitalassistants, or other processes and applications. Unfortunately, manyapps are largely black boxes that expose little or no information foruse by either the OS under which those apps are executed or for use byother third-party apps. As a result, valuable user information remainslocked up in individual app silos or are otherwise held private orsimply deleted by the apps in which that user information is generated,input, or otherwise selected.

Advantageously, various processes enabled by the Data Transfer Toolprovide a platform that enables users to take control of in-app data ofsource apps and share it effortlessly with one or more authorizeddestination apps. These processes also enable independent app developersto innovate on top of the availability of formatted user data byenabling their apps to receive information via subscription toapp-independent tasks.

As discussed herein, the Data Transfer Tool provides a data abstractionbased on the concept of tasks (e.g., actions and one or more associatedattributes). Tasks provide semantically meaningful information from rawtextual content extracted from arbitrary source apps for authorized useby the OS or other destination apps. More specifically, the DataTransfer Tool operates by applying the accessibility API, or assist APIor the like, to tap the interface between apps and the presentationsubsystem on computing devices. Raw text (and various controlinformation) being displayed by the source app is captured via theaccessibility API, or assist API or the like, along with optionalon-screen position information of the corresponding UI widget. Thisextracted information is then annotated with semantic labels via tasktemplates in the app model.

In various implementations, the app models may be generated through alearning process that models the extracted raw text using training datathat is generated either explicitly (e.g., from directed actionsperformed by human users interacting with the source app) or implicitly(e.g., from structured confirmation emails generated in response tocompleted transactions performed by users). The app models are thenapplied during runtime of the source app to convert the raw textextracted by the accessibility API, or assist API or the like, into astructured and semantically meaningful information stream comprisingapp-independent tasks. These tasks are then made available to otherauthorized apps either in real-time or from a local or remote task storeor database.

To assist third-party app developers to construct destination apps withsubscription capabilities for accessing and consuming tasks, in variousimplementations, the Data Transfer Tool provides a predefined “wrapperlibrary” or subscription API (also referred to herein as a “subscriptionmodule”) that can be called by the destination app to subscribe tospecific predefined tasks or task categories. In general, this wrapperlibrary or subscription API acts as an intermediary to provideauthorized destination apps with requested tasks that have beenextracted and stored to a local or remote task database or the like bythe Data Transfer Tool. Whenever a destination app receives a subscribedtask, the information (e.g., attributes) of that task is used topopulate or select one or more corresponding fields or controls of thedestination app.

In various implementations, destination apps interested in subscribingto specific tasks register with the Data Transfer Tool to request tasksubscription authorizations. In various implementations, during appsetup or installation or when an app is launched for the first time (oroptionally each time that an app is launched), the Data Transfer Toolmay notify the user of the tasks or task categories that the app wishesto subscribe to. The user can then either allow or deny the requestedpermissions or authorizations for the requested subscriptions, eitherglobally, or on a per-app basis.

Following user authorization to the subscription requests, the DataTransfer Tool monitors tasks extracted from source apps. Then, whenevera task for which a subscription exists becomes available, thecorresponding destination apps may be notified of the availability ofthe subscribed task. For example, in various implementations,destination apps are automatically invoked, initiated, or otherwiseexecuted by the Data Transfer Tool whenever a subscribed task becomesavailable. In other implementations, destination apps are automaticallynotified of the availability of subscribed tasks by the Data TransferTool whenever those apps are executed or otherwise invoked. In variousimplementations, authorized destination apps already executing mayreceive an automated notification from the Data Transfer Tool whenever asubscribed task or a task in a subscribed category becomes availablefrom any source app. Similarly, in various implementations, authorizeddestination apps may actively query the Data Transfer Tool to determinewhether a particular task (or task category) is available from anysource app for use by the authorized destination app.

To ensure that the user is in full control of his data, variousimplementations of the Data Transfer Tool provide both accesstransparency and access control, thereby improving user security. Forexample, in various implementations, at any point in time, accesstransparency is enabled via a UI that presents the user with a list orother identifying information that defines or specifies whichdestination apps are subscribing to particular tasks or task categories.Similarly, access control is enabled in various implementations via a UIthat presents the user with various mechanisms to update or change thesubscription authorizations or permissions of particular destinationapps to either allow or disallow access to particular tasks or taskcategories by those apps.

2.2 Information Extraction from Arbitrary Source Apps:

In order to extract in-app data, the Data Transfer tool taps into bothuser input (e.g., keyboard, touch, speech, gesture, pen, etc.) and thedisplay output of the OS and/or arbitrary applications.

For example, in various implementations the Data Transfer Tool applies alightweight OS instrumentation layer that taps into the presentationlayer (e.g., the UI) of the app in order to obtain a rich view of theuser's activities inside arbitrary apps. However, rather thaninstrumenting OS to tap user interaction with the UI of arbitrary apps,in other implementations, the Data Transfer Tool applies anaccessibility API, or assist API or the like, (available in most modernOSes) that enables features such as font and control size and colorchanges, audio narration of on-screen content, etc. The Data TransferTool leverages these types of APIs to tap on-screen content fromunmodified third-party source apps. In various implementations, the DataTransfer Tool includes a UI scraper module (see Section 2.2.1, below)that runs as a background service that applies the accessibility API, orassist API or the like, to scrape or extract content from any or allforeground apps.

In general, accessibility APIs, assist APIs, and the like areapplications, often provided as inherent OS accessibility functionality,that provides various mechanisms to improve user interaction witharbitrary apps for users who may have visual, physical or age-relatedlimitations that prevent them from fully seeing, hearing, or otherwiseinteracting with information being rendered by an application. As such,these types of APIs have direct access to content being rendered on adisplay in order to capture and reformat or re-present that content in away that may be more useful to particular users. For example, manyexisting accessibility APIs, assist APIs, and the like, recolor and/orresize text and/or controls or generate digital speech from text and/orcontrols (e.g., radio buttons, check boxes, drop-down lists, etc.) beingrendered on a display by an arbitrary app to assist users in interactingwith that app.

However, rather than apply the accessibility API, or assist API or thelike, to capture and reformat or re-present information in an app, theData Transfer Tool leverages the functionality of such APIs to recordthe text and UI widgets displayed on the screen, along with theirvarious properties, including, but not limited to, textual data, UIwidget position, activity ID (e.g., app page ID), whether and how theuser interacted with the widget (e.g., touch, swipe, keyboard entry,speech, gesture, etc.), information entered or selected by the user fora particular UI widget (e.g., user entered or selected attributevalues), timestamp of the interaction, etc. This information is thenprocessed by the Data Transfer Tool to derive a syntactic structure ofthe on-screen app content. Advantageously, use of the accessibility API,or assist API or the like, enables the Data Transfer Tool to extractcontent from arbitrary source apps without any modification to thosearbitrary apps.

As discussed in further detail in the following paragraphs, theextracted content is used to create app models for arbitrary sourceapps. These app models are then user in combination with informationextracted during subsequent user interaction with the correspondingsource app (again via the accessibility API, or assist API or the like)to provide semantically meaningful tasks (e.g., actions and one or morecorresponding attributes) to one or more destination applications thatare authorized to receive those tasks.

Further, multiple different kinds of tasks can typically be performed bythe user in any particular app. For example, a shopping app may supporttasks including, but not limited to, adding an item to a cart, deletingan existing item from the cart, entering or reading an item review, etc.Consequently, merely capturing or extracting the values or attributesset in an app (e.g., the content of various textboxes or UI controlstates or selections) may not be sufficient to determine what task isbeing performed. Therefore, in various implementations, the DataTransfer Tool also captures operations invoked by the user (e.g.,clicking an “add” vs. a “delete” button in a shopping app). Then, duringapp model inferencing, if the Data Transfer Tool is unable todistinguish between two or more tasks for the same application basedmerely on attribute values, the Data Transfer Tool further considers asequence of user interactions that may define a particular task. Suchsequences may be included along with the task attributes in the appmodel for the corresponding task.

2.2.1 Exemplary UI Scrapper Module Implementations:

In various implementations, the aforementioned UI scraper module isimplemented as a background user-level service that interfaces with theaccessibility API, or assist API or the like, to scrape content fromforeground apps. In various implementations, the user may grant or denypermission for the Data Transfer Tool to apply the UI scraper module toextract data from all apps or one or more individually specified apps.The UI scraper module then receives callbacks from the accessibilityAPI, or assist API or the like, whenever the API is active and receivingaccessibility events including, but not limited to touch, button click,gestures, changes in page content, speech inputs, etc., in response touser interaction with a particular source app. The UI scraper modulethen logs (e.g., stores) all the events of interest, together withadditional information including, but not limited to, an event ID, appID, values or attributes (e.g., user content entry or selection), screenposition of the corresponding UI widgets, etc., for which the UI scrapermodule receives a callback.

In various implementations, the use of computational resources isreduced by waiting to initiate the UI scraper until a source app fromwhich tasks may be extracted, and for which an app model is available,is launched. Similarly, the use of computational resources is alsoreduced by not processing the event log unless the user's computingdevice includes one or more subscribing destination apps that areauthorized to receive one or more of the tasks extracted from theforeground app. However, information may be extracted and stored forlater processing at such time that a corresponding app model may becomeavailable.

2.3 App-Specific Semantic Models:

Parsing user interactions with the UI of arbitrary apps may be complexbecause even apps from the same category (e.g., flight booking) oftendiffer substantially in their UI structure (e.g., the way various fieldsare named and laid out, how many app pages are utilized, etc.), how theapp acquires user input (e.g., keyboard, touch, speech, gestures, etc.),etc.

Given the limited screen real estate available to apps, especially onsmall or handheld computing devices (e.g., smartphones, tablets,notebooks, laptops, media players, eBooks, smartwatches, etc.), appdevelopers typically limit app UIs to a display of relevant information.Further, information in apps is typically presented in a structured waythat makes it easy for users to interact with the app. As such, theretends to be relatively little noise or extraneous information in thepresentation layer of most apps. Consequently, relatively simple appmodels can be constructed and applied in combination with theaccessibility API, or assist API or the like, to extract presentationlayer information from the app without requiring any OS or source appmodifications. Such capabilities and features facilitate deployment ofthe Data Transfer Tool on a wide range of computing platforms and OSesand increase the efficiency of interaction of the Data Transfer Toolwith arbitrary source apps.

Consequently, in various implementations, the Data Transfer Tool appliesan app-specific semantic model (also referred to herein as an app model)to information extracted from the source app via the accessibility API,or assist API or the like. The app model is applied to populateattributes of a structured task template that defines a particular useraction enabled by the source app. In various implementations, the DataTransfer Tool represents these attributes as generic key-valuecollections by providing a mechanism to tag data extracted from thepresentation layer of the app with semantically meaningful labels usingthe structured task templates.

In various implementations, the Data Transfer Tool applies variousnatural language processing techniques to app UIs to perform automatedextraction of app structure in order to automatically infer one or moreapp models. However, because of the enormous diversity in UI modalitiesin apps (even apps ostensibly directed to performing similar operationsor actions), it is generally more computationally efficient to provide aguided app interaction process for generating app models.

For example, one guided process employed by the Data Transfer Tooldirects users to point out and label semantically meaningful parts ofthe UI corresponding to predefined tasks. For example, the user may usea pointing device, gesture, speech-based input etc., to identify orotherwise label a location of one or more UI widgets of a source appalong with an identity or definition of a corresponding user action(e.g., book flight, enter blog text, select item, etc.). Thisuser-provided labeling, in combination with information extraction viathe accessibility API, or assist API or the like, provides the DataTransfer Tool with sufficient data for automatically inferring an appmodel for subsequent use with the corresponding source app.

A different guided process (see Section 2.3.1, below) directs users toperform one or more particular directed operations or actions using aparticular source app. As discussed in further detail in the followingparagraphs, such guided app interaction processes enable the DataTransfer Tool to efficiently infer app models in response to directedhuman interaction with a particular source app.

Regardless of the model construction technique used, the app modelconstruction process does not need to be performed or repeated by eachuser. In fact, once any app model has been created for any source appusing any modeling technique, that app model can be made available toany user that may use the corresponding source app. For example, invarious implementations, the Data Transfer Tool scans apps available ona user's computing device and then simply downloads one or morecorresponding app models from a store, database, or library ofpre-constructed app models. As such, any desired technique may beapplied to construct or otherwise define app models for use with theData Transfer Tool without departing from the intended scope of theconcepts and features described herein.

2.3.1 App Model Construction via Guided Human Interaction:

A wide variety of guided or directed human interactions with apps may beapplied by the Data Transfer Tool in order to match known (e.g.,directed) user interactions with content being selected or entered bythe user and then displayed or rendered via the presentation layer ofthe app. For example, in various implementations, a human or artificialintelligence (Al) expert defines one of more “task templates” for eachcategory (e.g., actions such booking a flight, adding items to ashopping cart, entering blog text, etc.), with placeholders for one ormore attributes. One or more users then each execute the directedactions conforming to the specified template, and select or enterappropriate attribute values of their choice. In variousimplementations, one or more of these users are then led through anautomatically constructed sequence of secondary tasks, each involvingexecuting the original task but with one or more of the attribute valueschanged.

In other words, given an expert-specified task template for a particularsource app or source app category, one or more predefined actions areautomatically presented for execution by one or more users. Informationextracted by the accessibility API, or assist API or the like, duringexecution of these predefined actions is then automatically related tothe data obtained from the presentation layer to match particular userentered or selected attributes with particular actions and UI widgets orvariables. In combination, this data is sufficient to enable the DataTransfer Tool to infer relationships between multiple actions andattributes associated with particular UI widgets of a source app forcompleting the task templates that are used to define the correspondingapp model.

In various implementations, the app model learning or inferencingprocess is performed by applying a differencing approach, wherein thedifferences in a UI state between two related task executions isattributed to differences in the attribute values in the correspondingtask instances. If any ambiguity remains, additional predefined taskinstances may be automatically generated and executed by one or moreusers in order to resolve the ambiguity.

Further, in various implementations, once an app model has beengenerated using such techniques, an app verification process may beperformed by directing one or more users to perform additionalpredefined actions using the app. This verification process thencontinues by comparing the results of those additional predefinedactions with results obtained from the app model. This verificationprocess helps to resolve issues in the app model resulting from humanerrors and/or spurious inferences from app models due to correlatedupdates to multiple UI widgets in the case that the source app isupdated. See Section 2.3.5 of this document for additional discussionregarding automated model updates in response source app changes orupdates.

More specifically, in various implementations, the Data Transfer Toolinitiates a guided human interaction process that instructs one or moreusers to perform one or more specific actions (e.g., book a flight, addan item to a shopping cart, add a review comment to a blog, initiate acommunications session, play a media file, interact with a homeautomation system, etc.) using a particular source app. The DataTransfer Tool then applies the accessibility API, or assist API or thelike, to log changes in the presentation layer of that app while eachuser performs each requested action. The Data Transfer Tool then appliesan inference engine module to infer the semantics of the in-app contentthat is displayed on the screen, i.e., the meaning of the various UIwidgets and relevant key-value pairs. For example, different “keys”could be used in different apps to refer to the same semanticinformation, e.g., the keys “origin”, “source”, and “dep city” may allbe used in different flight booking apps to refer to the same semanticentity (e.g., departure city). Sometimes, the key is implicit, such as akey denoted by an image (e.g., separate icons for adult and childpassenger counts), or even just screen position (e.g., the UI widget forthe departure city is generally placed to the left of that for thearrival city, without the need for either field to be labeled).

In other words, the Data Transfer Tool automatically evaluates loggedchanges in the presentation layer to infer a mapping from the particularactions to particular UI widgets (e.g., text entry fields, sliders,radio buttons, drop-down boxes, etc.) for entry or selection ofattributes and, optionally, additional information such as locations ofthose widgets on the display, timestamps, input method, etc. Each of theinferred mappings for a particular source app are then combined tocreate the overall app model for the particular source app.

In various implementations, the aforementioned guided human interactionprocess operates as a multi-stage process wherein the user is directedto execute a set of simple primary and secondary operations using thesource app. For example, an example of a primary operation may be: “Booka flight between any two cities on any date”. A subsequent secondaryoperation in the sequence may be: “Book a flight between the same twocities as before, but returning one day later.” In thisprimary/secondary operation example, the inference process performed bythe Data Transfer Tool would diff attribute values extracted from thepresentation layer in response to user performance of the primary andsecondary operations. In this primary/secondary scenario, it is likelythat a single UI widget value (corresponding to the return date) wouldbe different. Consequently, that widget would be labeled as “returndate” in the corresponding app model.

In general, when performing operations associated with primary tasks,users may enter or select any suitable attribute values for the specificprimary task instances that they execute when interacting with the UIwidgets. Similar tasks in different apps may call for different inputmethods, e.g., to set a date, some apps present a textbox for date entrywhile others present various forms of calendar widgets. Then, dependingon the primary task instance, for secondary or follow-on task instances,users may be directed to re-execute the task, with one or more attributevalues changed. By applying the accessibility API, or assist API or thelike, to observe changes to the overall presentation layer of the app UIwhile the user performs these primary and secondary tasks, the DataTransfer Tool can easily infer mappings from particular UI widgets(e.g., text entry fields, sliders, radio buttons, drop-down boxes, etc.)to particular actions relating to primary and secondary tasks. SeeSection 2.3.2, below, for a discussion of inferencing from this trainingdata. As noted above, the inferred mappings for a particular app arethen combined to create the overall app model for the particular app.

In other words, if the user is directed to change the contents of onetext entry field relating to a particular action, then the particularrelationship between that action and the corresponding text entry fieldcan be inferred and mapped by simply observing changes in thepresentation layer. Advantageously, this type of inference and mappingdoes not require the Data Transfer Tool to interpret or otherwiseunderstand the meaning of either the text entry or UI Widget selectionor the action being performed by the user. The observed relationshipbetween the text entry field or UI widget and the action is sufficientto enable the Data Transfer Tool to populate the task template for usein constructing the corresponding app model.

Table 1, below, illustrates various examples of task templates. Itshould be understood that the task templates illustrated in Table 1 areprovided for purposes of explanation and example are that the DataTransfer Tool is not intended to be limited to the use of theseexemplary task templates.

TABLE 1 Exemplary Task Templates for Various App Categories App CategoryTask Type Directed Actions (Attributes Denoted by < . . . >) FlightPrimary Search for a one-way flight from a <source city> to a<destination city> on a certain date for a certain <number ofpassengers> in a certain <class of travel> Secondary Search for aone-way flight using the same attribute values as in previous task butonly changing the <source city> Secondary Search for a one-way flightusing the same attribute values as in previous task but only changingthe <destination city> Secondary . . . Shopping Primary Browse for an<item> and add it to shopping cart Secondary Delete the <item> fromshopping cart Primary Add a <quantity of two> of a different <item> tothe shopping cart Secondary Delete the <items> from the shopping cartRecipe Primary Search for a <recipe> Secondary Search for a different<recipe> Transportation Primary Search for a route from a <sourcelocation> to a <destination> using a certain <mode of transport>Secondary Search for route using same attribute values as in theprevious task but only change the <source location> Secondary . . .Stock Primary Search for a <stock> quote Secondary Search for adifferent <stock> quote Primary Search for <stock> quote and add it to afavorites list Secondary Search for a different <stock> quote and add itto the favorites list Hotel Primary Search for hotel for a number of<guests> from a <start date> to an <end date> in a certain <city>Secondary Search for a hotel using the same attribute values as inprevious task but only changing the <guest> count Secondary . . .Weather Primary Search for weather conditions in a certain <city>Secondary Search for weather conditions in different <city> News PrimarySearch for a news <topic> Secondary Search for a different news <topic>Music Primary Search for songs by a certain <artist> and <genre>Secondary Search for songs using the same attribute values as inprevious task but only changing the <artist> Secondary . . . SocialPrimary Post a <status> on the social site Secondary Post a different<status> on the social site Primary Search for a <topic> on the socialsite Secondary Search for a different <topic> on the social siteCommunication Primary Post a <message> to a <contact> Secondary Post adifferent <message> to the same <contact>

Further, in various implementations, the Data Transfer Tool appliesvarious inferencing techniques during model training in ways that eitherenable inferencing using fewer secondary tasks, or, alternately, thatsimplify app model inference by maximizing the amount of informationgleaned from each secondary task. For example, in variousimplementations, the Data Transfer Tool considers an entity type of eachattribute in the task template, and allows multiple attributes to bechanged simultaneously in a secondary task, so long as they can bedisambiguated based on their entity types.

For example, during model training, the source airport and number ofpassengers may be changed simultaneously because passengers represent anumber that can be easily distinguished from an airport code or cityname. However, the source airport and destination airport may not besimultaneously changed since the entity type (e.g., city names orlocations) of both the source and destination airports are the same,thus making disambiguation more difficult. Further, even when the entitytype for two attributes is the same, in various implementations, theseattributes may be changed simultaneously so long as the change happensin a controlled manner that enables disambiguation (e.g., a secondarytask could involve incrementing a number of adults by 1 and a number ofchildren by a different amount, say 2. Similarly, as an example ofnon-numeric attributes, a secondary task could involve setting a sourcecity to the same value as the destination city in the primary task, andthe destination city to a new value.

Further, in various implementations, information extracted duringperformance of the directed actions is evaluated by the Data TransferTool to learn specific settings of attribute values supported by thecorresponding UI Widget of the app (e.g., complete set of airport codesin a drop-down list of the app). In various implementations, theselearned settings are optionally applied to run one or more verificationtask instances following app model construction. These verification taskinstances apply attribute values are drawn from the learned settings toverify consistency between the specified task and the inferences madefrom the corresponding task execution using the learned app model, andto update or correct the app model if necessary.

2.3.2 Inferencing from Training Task Data:

In general, guided training tasks, such as those discussed in Section2.3.1 of this document, generate raw UI traces (captured via theaccessibility API, or assist API or the like) comprising attributevalues of UI variables or widgets (e.g., textboxes or other UI controls)in the app. In various implementations, these raw UI traces are filteredand processed in multiple steps, as shown below in the inferencealgorithm of Table 2 to infer attribute to UI mappings based on thepredefined task templates. In particular, the exemplary pseudocodeillustrated in Table 2 provides an inference algorithm that maps eachattribute in the task template to corresponding UI variable(s) orwidgets of the source app representing actions performed by the user. Asnoted above, any desired technique may be applied to construct orotherwise define app models for use with the Data Transfer Tool withoutdeparting from the intended scope of the concepts described herein. Assuch, the Data Transfer Tool is not limited to the exemplaryinference-based modeling techniques illustrated in Table 2.

TABLE 2 Exemplary Inference Algorithm PseudocodeInfer_App_Model(UI_Trace) begin Num_Tasks = sizeof(UI_Trace);Num_Attributes = Num_Tasks − 1; for i = 0; i <= Num_Attributes; i + + doUI_Trace[i] = Filter_Out_Unnamed_Variables(UI_Trace[i]); UI_Trace[i] =Filter_Out_NonString_Variables(UI_Trace[i]); UI_Trace[i] =Filter_Out_All_But_Final_Value_Of_Variables(UI_Trace[i]); end for for i= 1; i <= Num_Attributes; i + + do UI_Diff_Set[i] = Diff(UI_Trace[i −1], UI_Trace[i]); i + +; end for UI_Diff_Set =Filter_Out_Variables_With_Repeated_Occurences(UI_Diff_Set); App_Model =UI_Diff_Set; end

In general, the exemplary inference algorithm illustrated in Table 2processes the raw UI traces (e.g., “UI_Trace” captured via theaccessibility API, or assist API or the like) from a user's directedinteraction with the app. As illustrated in Table 2, in variousimplementations, the inference algorithm first filters out any UIvariables from each UI trace for which the app developer has notassigned a name. The rationale for this filtering step is that for therelatively small subset of app variables that may be relevant to appmodel construction, app developers typically do assign a variable name.

In addition, as illustrated in Table 2, in various implementations, theinference algorithm filters out any non-string variables, e.g., any UIvariables whose values are not text strings. The rationale for thisfiltering step is that it has been observed that restricting tovariables with textual values was unlikely to eliminate relevantvariables in typical apps for purposes of app model construction. Forexample, even UI widgets such as radio buttons tend to have textualvalues (e.g., “Economy”, “Business”, etc., in the context of class oftravel). For controls such as check boxes, radio buttons, etc., thestate of each control may be recorded via UI traces captured by theaccessibility API, or assist API or the like, to determine which textualattribute value was selected via the control.

Further, in various implementations, each variable's value is logged inthe raw UI trace captured by the accessibility API, or assist API or thelike, whenever the value changes. A particular variable may occurmultiple times in the UI trace (e.g., the user might set the passengercount to 1 and then change it to 2, before selecting “submit” or somecorresponding UI control). However, while such changes may be relevantto a particular user interaction, such changes are not typicallyrelevant to the inferred mapping between attributes and UI variable(s)of the source app for purposes of inferring the app model. Consequently,as illustrated by the inference algorithm in Table 2, all but the finalvalue of each variable may be filtered out without reducing the accuracyof the inferred app model.

Next, also as illustrated in Table 2, With respect to the set of tracescorresponding to the primary task and the subsequent secondary tasks,for each pair of successive tasks (which may correspond to the userexecuting a task and then the same task again, but with a change in thesetting of one attribute), the inference algorithm computes thedifference, e.g., UI_Diff_Set, comprising the variables whose valueschanged across the pair of successive tasks. Consequently, thisdifference set represents a candidate set of UI variables that maycorrespond to a particular attribute i.

As illustrated in Table 2, UI_Diff_Set is then further refined to createthe app model by filtering out candidate variables that appear in thedifference set for multiple attributes. These types of repeatedoccurrences of attributes suggests that the changes occurring in thevariables from one task to the next are likely not due to an intentionalchange by the user of the changed attribute value. For example, avariable corresponding to the current time will periodically orcontinuously change in value between successive tasks, and hence willoccur in the difference set of all attributes. However, such repeatedchanges are clearly not due to changes in the attribute value beingentered or selected by the user. Consequently, as illustrated in Table2, candidate UI variables evidencing continuous or periodic changes arefiltered out of the difference set. As illustrated in Table 2, followingthe final filtering of the UI_Diff_Set, the inference algorithm outputsthe refined difference set as “App_Model” which comprises thosevariables that are likely to be related to the corresponding attributes.The resulting app model can then be provided for use by other users.

However, it is possible that the resulting app model contains one ormore incorrect mappings due to human error, e.g., the primary orsecondary task (i.e., the directed actions) may have asked the user tochange the source airport but the user inadvertently changed some othervariable instead. To protect against such errors, in variousimplementations, the inferred app model may be tested against additionaldirected actions by a small subset of users. For example, this subset ofusers may be directed to execute additional training tasks for each appto determine whether there is a majority agreement between the inferredapp model and new app models that are inferred for each of the subset ofusers in response each user's interaction with the app. For instance,with three users, if a certain variable v appears in the difference setfor attribute i in the traces from at least two users, that variable isretained in the final app model, otherwise, it is filtered out. Afterany app model for a particular version of an app has been finalized, itmay be provided to other users for use with local instances of the DataTransfer Tool. In other words, other users can simply consume thefinalized version of the app model without participating in any of themodel training phases involving directed actions and model inference.

Further, in various implementations, the Data Transfer Tool considersaliasing of attributes. In particular, the inferred app model maps eachattribute to the corresponding UI variable(s) of an app. In most cases,an attribute maps onto just one variable or UI widget. However, in somecases, the attribute may map onto multiple variables, which is termed“aliasing.” For example, when a user executes the task of adding a newitem to the cart in a shopping app, a textbox corresponding to the itemname and another textbox corresponding to the price might both change.Furthermore, for the small number of training tasks executed by users,the values appearing in both of these textboxes might be unique. In suchcases, the above-described inference algorithm would not be able to tellwhich of the two textboxes constitutes an appropriate description of theitem. In these cases, the Data Transfer Tool directs one or more usersto perform one or more verification tasks to help refine the app model.For example, a user may be directed to perform an action to input orselect an attribute (e.g., a variable value) relative to a particular UIcontrol to validate a particular task of the app model.

2.3.4 Leveraging Implicit Training Data for Model Construction:

In the preceding discussion, training data was generated explicitly byhaving human users execute particular tasks (e.g., one or morepredefined actions to enter or select attributes that were then mappedto particular UI controls or variables). However, as noted above,training data may also be obtained implicitly. For example, a usersession with a particular app that involves a completed transactionoften results in an external manifestation such as a confirmation email.Such confirmation emails, conforming to a standardized format (e.g.,schema.org email templates), are used by a variety of online servicesand apps. In various implementations, the Data Transfer Tool applies thestandardized format to match information extracted from the confirmationemail with information obtained from the UI traces captured during userinteraction with the app. This provides another mechanism that enablesthe Data Transfer Tool to learn or infer the mappings that definesemantics of the in-app content. Further, implicit training data may becombined with explicit training data to construct or validate app modelsinferred using any desired modeling technique.

As with the explicit generation of training data discussed above, if anyambiguity remains after the email to UI trace matching step, e.g.,because the same value occurs multiple times in a transaction (e.g., 2passengers travelling on 2 July), data from confirmation emails foradditional transactions, whether by the same user or a different user,can help with disambiguation. Once the mapping between UI elements andthe components of the structured confirmation email has beenestablished, the Data Transfer Tool can then apply the resulting appmodel to extract the semantics of in-app information resulting from userinteraction with the app, even when that interaction does not result ina completed transaction (and hence no confirmation email).

2.3.5 Model Updates for New App Versions:

Apps are often updated for any of a number of reasons, e.g., add orremove features, change UI structure, fix bugs, change variable names,etc. Depending on the changes made to an app, the corresponding appmodel may decrease in accuracy or become obsolete. Such issues areaddressed by various implementations of the Data Transfer Tool byleveraging a static app binary checking process in combination with UIwidget position and value-based matching to automatically update appmodels when corresponding source apps are updated.

For example, in various implementations, the Data Transfer Tool performsan automated static analysis of binaries of apps to determine whethervariable names were added or removed relative to the variables includedin the app model for the prior version of the app. In cases where mostor all of the variable names in the existing app model also exist in thebinary of the new app version, it is likely that the existing app modeland the corresponding mapping of attributes to variable names remainsvalid across multiple versions of the app.

As noted above, in various implementations, the Data Transfer Tooloptionally applies the accessibility API, or assist API or the like, tocapture on-screen position information of the UI widgets of the app whencapturing UI traces for use in constructing app models. Consequently, invarious implementations, the Data Transfer Took leverages this UI widgetposition information in combination with corresponding key-value pairsin the app model to validate one or more of the tasks of the app modelagainst the new or updated app version.

For example, the Data Transfer Tool can compare a normalized x, ylocation of a UI widget on the screen and a set of values observed (viaUI traces) for the key during actual user interaction with the app.Then, when an app update occurs, if the developer changes the variablenames (key names) and/or location of the corresponding UI widget on thescreen, the Data Transfer Tool looks for the nearest key in a spatialand value domain to find the new key corresponding to the taskattribute. This information is then applied to update the existing appmodel to correspond to the new app version. While this approach is notalways 100 percent accurate, it has been observed to significantlyreduce errors in the app model relative to new app versions. Further,whatever errors remain in the app model after this process may then becorrected using any combination of the explicit or implicit errorcorrection and training techniques described above.

2.4 Exemplary Cross-App Sharing Apps:

As noted above, the tasks extracted by the Data Transfer Tool fromarbitrary source apps may be provided to one or more authorizedsubscribing apps. However, because these extracted tasks may also bestored for later use, those tasks can be indexed, sorted, and otherwiseprocessed to enable a wide variety of new apps and services thatsubscribe to those tasks. The following paragraphs describe fiveexamples of some of the many new apps and services that may beconstructed based on the availability of stored or logged tasks and onreceipt of real time task notifications in response to tasksubscriptions. These five exemplary apps are based on a mashup of theuser's data de-siloed from various travel and messaging apps by creatingvarious apps that subscribe to tasks that are extracted from thosetravel and messaging apps during user interaction. Uses of the stored orlogged tasks provided by the Data Transfer are clearly not limited tothe examples described below or to interaction with travel or messagingapps.

For example, some of the new cross-app data sharing apps enabled by thesubscription to tasks extracted from the Data Transfer Tool include, butare not limited to:

-   -   1) A history app that collates and presents an interactive        timeline of user activities across task categories derived from        multiple apps, including, but not limited to, shopping, movies,        travel, messaging, etc.    -   2) A travel alert app that automatically schedules price alerts        for flights (or other travel modes) searched for (but not        booked) by the user.    -   3) A limousine service app that automatically launches the        user's preferred limo or cab app with pick-up time based on        flight departure (or arrival) and the drop-off (or pick-up)        location set to the appropriate airport when a flight is booked.    -   4) A smart messaging app that presents a unified view of users        and messages combined from different messaging apps.    -   5) An in-app assistant app that runs as a background service and        provides pop-ups or other alert types relative to any arbitrary        source app with which the user is currently interacting.

2.4.1 Exemplary History App:

In various implementations, the exemplary history app constructs andpresents a unified timeline of activities that have been performedinside multiple different apps, regardless of whether there is anyrelation or interaction capabilities between those multiple apps. Thehistory app constructs this unified task timeline by subscribing tomultiple task categories, including, but not limited to, shopping,movies, cuisine search, flight, hotel, rental car bookings, messaging,etc. The history app provides a search feature to index into tasks usingfilters such as time, keywords and task type.

2.4.2 Exemplary Travel Alert App:

In various implementations, the exemplary travel alert app subscribes tovarious travel search and travel booking events extracted from multipletravel apps and then creates a price alert to inform the user of changesin travel prices. Travel prices may be obtained by the travel alert appby automatically periodically querying one or more travel sites orservices to determine travel prices using travel modes corresponding tothe user travel search events.

For example, in the context of airline flights, the exemplary travelalert app may subscribe to “flight search” and “flight booking” eventsfrom multiple flight apps. Then, when the travel alert app receives“flight search” events for particular origin and destination airports(on potentially different dates), but does not receive a “flightbooking” notification or task, it can be presumed that the user may bechecking flight prices. Consequently, the travel alert app initiates analert process that automatically generates a price change alert ornotification for the range of dates seen in the travel search events.

In various implementations, the travel alert app may either present thisalert or notification directly to the user, or it can generate an alerttask that is available for subscription by other apps, such as amessaging app for example. In either case, if the price alert triggerswith a price change in excess of a predefined or user-specifiedthreshold, e.g., 10% higher or lower than the price seen from the searchevents, the travel alert app may then alert or notify the user. Invarious implementations, if the travel alert app subsequently receives a“flight booking” event relating to the flight search events, the alertprocess may be dismissed, thereby terminating future price changealerts.

2.4.3 Exemplary Limousine Service App:

In various implementations, the limousine service app subscribes totravel itinerary tasks (e.g., flight event events, calendar events,etc.). Further, in various implementations, the limousine service appinfers the user's home and office address using location data atdifferent times of day, or current GPS location at the time that alimousine or cab may be desired by the user. Then, when the user books aflight (or other travel mode) from some location, the limousine serviceapp provides a single-touch limousine or cab booking that launches theuser's preferred limo or cab app with numerous prepopulated fields(which may be changed or updated by the user) based on tasksubscriptions of the limousine service app.

For example, based on task subscriptions, the limousine service app mayprepopulate various fields including, but not limited to: name, numberof passengers, pick-up location set to user home or office address(depending on the day of week and time of the flight) or currentlocation; destination address or location automatically set to theairport (or other departure location such as a train station); pick-uptime set based on travel time estimates between home/office and theairport. Similarly, in various implementations, the limousine serviceapp may provide similar single-touch reservations to book a limousine orcab between the arrival airport and hotel (if a hotel booking event orsubscribed task has been received by the limousine service app) and arental-car booking event or subscribed task is not seen. Similarly, invarious implementations, the limousine service app provides single-touchreservations to book a limousine or cab between the original departureairport and the user's home or office (or other location) based on theuser's scheduled return flight.

2.4.4 Exemplary Smart Messaging App:

In various implementations, the smart messaging app subscribes tomessaging events or tasks, which trigger whenever a new message is sentto or received from any of the messaging apps utilized by the user (orsome user specified subset of those messaging apps). The Data TransferTool then infers the messaging app name (i.e., which messaging app sentor received a message), user name or messaging handle or ID, and themessage content whenever a messaging event is triggered via sent orreceived messages. In various implementations, the smart messaging appthen constructs and presents a UI-based mashup of all messaging appsutilized by the user, wherein a combined view of all of the user'smessages can be viewed in one place. In addition, in variousimplementations, for apps that support deep linking (e.g., Skype®), thesmart messaging app may enable user selection of a particular message tolaunch the corresponding app and directly navigate to the appropriatepage in that app so that the user may continue his messaging in thatapp, if he so desires.

2.4.2 In-App Assistant App:

In various implementations, the in-app assistant app runs as abackground process that subscribes to the tasks (e.g., user entered orselected data) being extracted in real-time from an arbitrary app withwhich the user is interacting. The in-app assistant app then retrievesand formats a relevant response from a local or remote data source inresponse to a search or query based on the content of subscribed task.The in-app assistant app then present the relevant response in real-timevia any alert or notification technique, including, but not limited to,pop-ups, overlays, speech-based alerts, etc. This response by the in-appassistant app presents relevant information to the user based on thecontent of the subscribed tasks.

For example, assume that the user is communicating via a messaging-typesource app for which an app model exists. If the user enters or receivesa message via that source app regarding a particular topic, the DataTransfer Tool will automatically generate a corresponding taskcomprising a message sent or received action having attributes relatingto the particular topic (e.g., movie, recipe, current news event, music,celebrity personality, weather, etc.). Assuming that the in-appassistant is authorized and subscribed to receive that task, the in-appassistant will then automatically generates a response (e.g., pop-up,overlay, speech, etc.) that provides real-time information relevant tothe particular topic identified in that task. For example, assuming thatthe task references a particular movie, the in-app assistant maygenerate real-time responses (via internet searches or other searches orqueries) including but not limited to, local movie theaters showing themovie with times and prices, movie reviews, etc.

Advantageously, the in-app assistant app improves user interactionexperience by providing information to the user that is relevant toreal-time actions and attributes in arbitrary source apps while the useris interacting with source apps without in any way modifying the sourcecode of that arbitrary source app. As such, the in-app assistant appimproves the functionality of arbitrary source apps by presentingrelevant information to users regardless of the structure or features ofthe arbitrary source app.

In other words, the in-app assistant app may run constantly in thebackground, and have subscriptions to some or all of the tasks scrapedor extracted from some or all of the arbitrary source apps that havebeen modeled. The in-app assistant app then provides relevantinformation whenever a task or informational item is extracted from somearbitrary source app by the Data Transfer Tool and provided as asubscribed task to the in-app assistant. As such, the in-app assistantapp augments existing unmodified source apps in response to informationextracted from those unmodified source apps.

In various implementations, the response provided by the in-appassistant app is user selectable. As such, the user can optionally clickon or otherwise select or respond to the information being presented bythe in-app assistant app, e.g., click on a hyperlink “Movie Times” tosee, hear, or otherwise interact with a list of nearby movie timespresented in response to mention of a particular movie in a messagingapp. User input either in the arbitrary source apps, or in response toany information presented to the user by the in-app assistant app, canbe textual, interaction with GUI controls, gesture-based, speech-based,or any combination of such input modalities.

In various implementations, the in-app assistant app suggestsinformation to be entered into third-party apps that are not themselvesmodified to subscribe to tasks extracted by the Data Transfer Tool. Inthis case, an app model created for that third-party app will enable thein-app assistant app to determine what information is to be entered orselected for a particular UI widget of the third-party app. The in-appassistant app may then check to see whether it has any tasks matchingthat UI widget. If so, the in-app assistant app can initiate a pop-up orthe like suggesting the information to be entered or selected. Further,in various implementations, the in-app assistant app automaticallycopies relevant attributes of the task to the OS or app clipboard sothat the user can simply past the suggested information into the UIwidget of that third-party app.

2.5 Additional Implementations of the Data Transfer Tool:

The following paragraphs briefly introduce a variety of additional usesand implementations of the Data Transfer Tool. Each of the followingexemplary uses and implementations are intended to be understood in viewof the preceding discussion.

In various implementations, the Data Transfer Tool models user behaviorswith respect to various applications. For example, if the Data TransferTool observes that particular sequences of tasks are generated bydifferent source apps, the Data Transfer Tool may use those observationsto automatically initiate tasks or suggest actions. For example, if auser is observed (via generation of tasks) to frequently initiate a cabreservation app following a flight booking app, the Data Transfer canautomatically initiate the cab reservation app (or ask the user if thatapp should be initiated) as soon as the user books a flight. In otherwords in various implementations, the data Transfer Tool mayautomatically initiate one or more third-party apps in response to userinteraction with other source apps. In addition, the Data Transfer Toolmay automatically populate one or more data entry or selection fields ofthe automatically initiated app based on one or more tasks scrapped fromone or more modeled source apps with which the user has beeninteracting.

In various implementations, the Data Transfer Tool mines in-appinformation from arbitrary source apps that is obtained by scrapingcontent from the arbitrary source app to model user behaviors (e.g.,ordering pizza every Friday night, searching particular topics, joiningor participating in various organizations or blogs, etc.). These modeleduser behaviors are then applied by the Data Transfer Tool toautomatically customize one or more third-party apps or digitalassistants.

Relevant sponsored content and/or sponsored apps are presented byvarious implementations of the Data Transfer Tool (or the aforementionedin-app assistant app) in response to information scraped from modeledsource apps by the Data Transfer Tool.

In cases that a particular user interacts with multiple computingdevices, the Data Transfer Tool can scrape information from source appsrun on one of those computing devices and provide corresponding tasks toapps running on one or more of the user's other devices. This type ofdata transfer can be accomplished using any desired wired or wirelesscommunications mechanism.

Similarly, devices of additional users (e.g., family members, friends,business associates, etc.) may be authorized (on a per-device level, aper-user-level, or a per-task level) to receive data scraped from anysource application of a particular user. This type of data transfer canbe accomplished using any desired wired or wireless communicationsmechanism.

In various implementations, anonymized scrapped data may be provided toarbitrary users. For example, if a large number of users that interactwith “app A” subsequently interact with “app B,” or perform a specificaction (e.g., go to a particular restaurant in Seattle after booking aflight to Seattle), the Data Transfer Tool (via the aforementionedin-app assistant) can suggest that users interacting with “app A” alsoexecute “app B” (or perform the action such as going to the restaurantin Seattle). In various implementations, the Data Transfer Tool requestsuser permission prior to including their anonymized app usage or actionsas recommendations to other users.

In various implementations, the Data Transfer Tool scrapes informationfrom arbitrary apps, whether or not those apps are modeled, and can thencopy or paste some or all of the scraped information to any desiredtext, note, word processing app, email program, messaging app, etc. Forexample, a user-accessible data store that includes either or bothstructured data (e.g., tasks) and unstructured data (e.g., raw scrapeddata) can be selected and copied as a group or on an element-by-elementbasis into arbitrary apps.

The Data Transfer Tool also enables a variety of smart messaging apps.For example, the Data Transfer Tool can, over time, can learn apreferred messaging app used by a user for each of a plurality ofdifferent contacts. The Data Transfer Tool can then automaticallyinitiate the preferred communications app for a particular contactwhenever the Data Transfer Tool scrapes data relating to that contactfrom any app with which the user is interacting. For example, if theuser clicks on a contact name in an any arbitrary app, that userinteraction is scrapped by the Data Transfer Tool, which then initiatesa communications app that has been modeled/learned as being thepreferred app for that particular contact. In other words, the DataTransfer Tool can initiate particular third-party apps in response toinformation scrapped from any arbitrary app with which the user isinteracting.

In various implementations, the Data Transfer Tool automatically movesor copies information from one application to another to act as a bridgebetween those applications. For example, assume that the user prefers afirst communications app to a second communications app, and the contacthe is messaging with prefers the second communications app. In thiscase, the user can run both apps, and the Data Transfer Tool can scrapemessages received by the second communications app from the contact andpopulate the first communications app with those scrapped messages.Similarly, the Data Transfer Tool can scrape user-entered messages fromthe first app and populate those messages into the second communicationsapp for transmission to the contact. As a result, the user cancommunicate with the contact in real-time even if the user and thecontact are each using different and incompatible messaging apps.

3.0 Operational Summary of the Data Transfer Tool:

The processes described above with respect to FIG. 1 and FIG. 2, and infurther view of the detailed description provided above in Sections 1and 2, are illustrated by the general operational flow diagrams of FIG.3, FIG. 4 and FIG. 5. In particular, FIG. 3, FIG. 4 and FIG. 5 providean exemplary operational flow diagrams that summarize the operation ofsome of the various implementations of the Data Transfer Tool. FIG. 3,FIG. 4 and FIG. 5 are not intended to provide exhaustive representationsof all of the various implementations of the Data Transfer Tooldescribed herein, and the implementations represented in FIG. 3, FIG. 4and FIG. 5 are provided only for purposes of explanation.

Further, any boxes and interconnections between boxes that may berepresented by broken or dashed lines in any of FIG. 3, FIG. 4 and FIG.5 represent optional or alternate implementations of the Data TransferTool described herein, and any or all of these optional or alternateimplementations, as described below, may be used in combination withother alternate implementations that are described throughout thisdocument.

In general, as illustrated by FIG. 3, in various implementations, theData Transfer Tool begins operation by applying (300) an accessibilityAPI, or assist API or the like, to extract a plurality of data itemsfrom text being rendered on a display device in response to userinteraction with a source application. In various implementations, theData Transfer Tool then applies (310) a predefined application contentmodel corresponding to the source application to identify and label theextracted data items. In various implementations, the Data Transfer Toolthen applies (320) a subscription process to authorize and subscribe oneor more destination applications to access the identified and labeledextracted data items. Finally, in various implementations, the DataTransfer Tool automatically applies (330) one or more of the identifiedand labeled extracted data items to populate one or more correspondingdata fields of one or more of the authorized destination applications.

In general, as illustrated by FIG. 4, in various implementations, theData Transfer Tool begins operation by applying (400) an accessibilityAPI, or assist API or the like, to extract a plurality actions andcorresponding attributes from content and UI controls being rendered ina presentation layer of a source app. In various implementations, theData Transfer Tool then applies (410) a content model of the source appto format the extracted actions and corresponding attributes asapp-independent tasks. Finally, in various limitations, the DataTransfer Tool applies (420) one or more of the app-independent tasks topopulate one or more variables of one or more of destination apps thatare subscribed to receive those app-independent tasks.

In general, as illustrated by FIG. 5, in various implementations, anin-app assistant app of the Data Transfer Tool begins operation byscraping (500) user entered information from a presentation layer of anarbitrary application. In various implementations, the in-app assistantapp of the Data Transfer Tool then performs (510) an automated search toretrieve information relevant to the scraped information. Finally, invarious implementations, the in-app assistant app of the Data TransferTool then presents (520) a notification in an assistant app that isunrelated to the arbitrary app to present the information relevant tothe scraped information

4.0 Exemplary Implementations of the Data Transfer Tool:

The following paragraphs summarize various examples of implementationsthat may be claimed in the present document. The implementationssummarized below are not intended to limit the subject matter that maybe claimed in view of the detailed description of the Data TransferTool. Further, any or all of the implementations summarized below may beclaimed in any desired combination with some or all of theimplementations described throughout the detailed description and anyimplementations illustrated in one or more of the figures, and any otherimplementations and examples described below. The followingimplementations and examples are intended to be understood in view ofthe detailed description and figures described throughout this document.

In various implementations, a Data Transfer Tool is implemented bymeans, processes or techniques for extracting, labeling and storing userdata siloed by a source application so that the stored data can beshared across and between a plurality of authorized destinationapplications to improve user interaction efficiency with the authorizedapplications.

As a first example, in various implementations, a computer-implementedprocess is implemented via means, processes or techniques for applyingan API, such as, for example, an accessibility API, or assist API or thelike, to extract a plurality of data items from text being rendered on adisplay device in response to user interaction with a sourceapplication. This computer-implemented process then applies a predefinedapplication content model corresponding to the source application toidentify and label the extracted data items. In various implementations,a subscription or other authorization process is applied to authorizeand subscribe one or more destination applications to access theidentified and labeled extracted data items. The computer-implementedprocess then automatically applies one or more of the identified andlabeled extracted data items to populate one or more corresponding datafields of one or more of the authorized destination applications.

5.0 Exemplary Operating Environments:

The Data Transfer Tool implementations described herein are operationalwithin numerous types of general purpose or special purpose computingsystem environments or configurations. FIG. 6 illustrates a simplifiedexample of a general-purpose computer system on which variousimplementations and elements of the Data Transfer Tool, as describedherein, may be implemented. Any boxes that are represented by broken ordashed lines in the simplified computing device 600 shown in FIG. 6represent alternate implementations of the simplified computing device.As described below, any or all of these alternate implementations may beused in combination with other alternate implementations that aredescribed throughout this document.

The simplified computing device 600 is typically found in devices havingat least some minimum computational capability such as personalcomputers (PCs), server computers, handheld computing devices, laptop ormobile computers, communications devices such as cell phones andpersonal digital assistants (PDAs), multiprocessor systems,microprocessor-based systems, set top boxes, programmable consumerelectronics, network PCs, minicomputers, mainframe computers, and audioor video media players.

To allow a device to realize the Data Transfer Tool implementationsdescribed herein, the device should have a sufficient computationalcapability and system memory to enable basic computational operations.In particular, the computational capability of the simplified computingdevice 600 shown in FIG. 6 is generally illustrated by one or moreprocessing unit(s) 610, and may also include one or more graphicsprocessing units (GPUs) 615, either or both in communication with systemmemory 620. The processing unit(s) 610 of the simplified computingdevice 600 may be specialized microprocessors (such as a digital signalprocessor (DSP), a very long instruction word (VLIW) processor, afield-programmable gate array (FPGA), or other micro-controller) or canbe conventional central processing units (CPUs) having one or moreprocessing cores and that may also include one or more GPU-based coresor other specific-purpose cores in a multi-core processor.

In addition, the simplified computing device 600 may also include othercomponents, such as, for example, a communications interface 630. Thesimplified computing device 600 may also include one or moreconventional computer input devices 640 (e.g., touchscreens,touch-sensitive surfaces, pointing devices, keyboards, audio inputdevices, voice or speech-based input and control devices, video inputdevices, haptic input devices, devices for receiving wired or wirelessdata transmissions, and the like) or any combination of such devices.

Similarly, various interactions with the simplified computing device 600and with any other component or feature of the Data Transfer Tool,including input, output, control, feedback, and response to one or moreusers or other devices or systems associated with the Data TransferTool, are enabled by a variety of Natural User Interface (NUI)scenarios. The NUI techniques and scenarios enabled by the Data TransferTool include, but are not limited to, interface technologies that allowone or more users user to interact with the Data Transfer Tool in a“natural” manner, free from artificial constraints imposed by inputdevices such as mice, keyboards, remote controls, and the like.

Such NUI implementations are enabled by the use of various techniquesincluding, but not limited to, using NUI information derived from userspeech or vocalizations captured via microphones or other input devices640 or system sensors 605. Such NUI implementations are also enabled bythe use of various techniques including, but not limited to, informationderived from system sensors 605 or other input devices 640 from a user'sfacial expressions and from the positions, motions, or orientations of auser's hands, fingers, wrists, arms, legs, body, head, eyes, and thelike, where such information may be captured using various types of 2Dor depth imaging devices such as stereoscopic or time-of-flight camerasystems, infrared camera systems, RGB (red, green and blue) camerasystems, and the like, or any combination of such devices. Furtherexamples of such NUI implementations include, but are not limited to,NUI information derived from touch and stylus recognition, gesturerecognition (both onscreen and adjacent to the screen or displaysurface), air or contact-based gestures, user touch (on varioussurfaces, objects or other users), hover-based inputs or actions, andthe like. Such NUI implementations may also include, but are not limitedto, the use of various predictive machine intelligence processes thatevaluate current or past user behaviors, inputs, actions, etc., eitheralone or in combination with other NUI information, to predictinformation such as user intentions, desires, and/or goals. Regardlessof the type or source of the NUI-based information, such information maythen be used to initiate, terminate, or otherwise control or interactwith one or more inputs, outputs, actions, or functional features of theData Transfer Tool.

However, it should be understood that the aforementioned exemplary NUIscenarios may be further augmented by combining the use of artificialconstraints or additional signals with any combination of NUI inputs.Such artificial constraints or additional signals may be imposed orgenerated by input devices 640 such as mice, keyboards, and remotecontrols, or by a variety of remote or user worn devices such asaccelerometers, electromyography (EMG) sensors for receiving myoelectricsignals representative of electrical signals generated by user'smuscles, heart-rate monitors, galvanic skin conduction sensors formeasuring user perspiration, wearable or remote biosensors for measuringor otherwise sensing user brain activity or electric fields, wearable orremote biosensors for measuring user body temperature changes ordifferentials, and the like. Any such information derived from thesetypes of artificial constraints or additional signals may be combinedwith any one or more NUI inputs to initiate, terminate, or otherwisecontrol or interact with one or more inputs, outputs, actions, orfunctional features of the Data Transfer Tool.

The simplified computing device 600 may also include other optionalcomponents such as one or more conventional computer output devices 650(e.g., display device(s) 655, audio output devices, video outputdevices, devices for transmitting wired or wireless data transmissions,and the like). Typical communications interfaces 630, input devices 640,output devices 650, and storage devices 660 for general-purposecomputers are known to those skilled in the art, and will not bedescribed in detail herein.

The simplified computing device 600 shown in FIG. 6 may also include avariety of computer-readable media. Computer-readable media can be anyavailable media that can be accessed by the computing device 600 viastorage devices 660, and include both volatile and nonvolatile mediathat is either removable 670 and/or non-removable 680, for storage ofinformation such as computer-readable or computer-executableinstructions, data structures, program modules, or other data.

Computer-readable media includes computer storage media andcommunication media. Computer storage media refers to tangiblecomputer-readable or machine-readable media or storage devices such asdigital versatile disks (DVDs), blu-ray discs (BD), compact discs (CDs),floppy disks, tape drives, hard drives, optical drives, solid statememory devices, random access memory (RAM), read-only memory (ROM),electrically erasable programmable read-only memory (EEPROM), CD-ROM orother optical disk storage, smart cards, flash memory (e.g., card,stick, and key drive), magnetic cassettes, magnetic tapes, magnetic diskstorage, magnetic strips, or other magnetic storage devices. Further, apropagated signal is not included within the scope of computer-readablestorage media.

Retention of information such as computer-readable orcomputer-executable instructions, data structures, program modules, andthe like, can also be accomplished by using any of a variety of theaforementioned communication media (as opposed to computer storagemedia) to encode one or more modulated data signals or carrier waves, orother transport mechanisms or communications protocols, and can includeany wired or wireless information delivery mechanism. The terms“modulated data signal” or “carrier wave” generally refer to a signalthat has one or more of its characteristics set or changed in such amanner as to encode information in the signal. For example,communication media can include wired media such as a wired network ordirect-wired connection carrying one or more modulated data signals, andwireless media such as acoustic, radio frequency (RF), infrared, laser,and other wireless media for transmitting and/or receiving one or moremodulated data signals or carrier waves.

Furthermore, software, programs, and/or computer program productsembodying some or all of the various Data Transfer Tool implementationsdescribed herein, or portions thereof, may be stored, received,transmitted, or read from any desired combination of computer-readableor machine-readable media or storage devices and communication media inthe form of computer-executable instructions or other data structures.Additionally, the claimed subject matter may be implemented as a method,apparatus, or article of manufacture using standard programming and/orengineering techniques to produce software, firmware 625, hardware, orany combination thereof to control a computer to implement the disclosedsubject matter. The term “article of manufacture” as used herein isintended to encompass a computer program accessible from anycomputer-readable device, or media.

The Data Transfer Tool implementations described herein may be furtherdescribed in the general context of computer-executable instructions,such as program modules, being executed by a computing device.Generally, program modules include routines, programs, objects,components, data structures, and the like, that perform particular tasksor implement particular abstract data types. The Data Transfer Toolimplementations may also be practiced in distributed computingenvironments where tasks are performed by one or more remote processingdevices, or within a cloud of one or more devices, that are linkedthrough one or more communications networks. In a distributed computingenvironment, program modules may be located in both local and remotecomputer storage media including media storage devices. Additionally,the aforementioned instructions may be implemented, in part or in whole,as hardware logic circuits, which may or may not include a processor.

Alternatively, or in addition, the functionality described herein can beperformed, at least in part, by one or more hardware logic components.For example, and without limitation, illustrative types of hardwarelogic components that can be used include field-programmable gate arrays(FPGAs), application-specific integrated circuits (ASICs),application-specific standard products (ASSPs), system-on-a-chip systems(SOCs), complex programmable logic devices (CPLDs), and so on.

6.0 Other Implementations:

The foregoing description of the Data Transfer Tool has been presentedfor the purposes of illustration and description. It is not intended tobe exhaustive or to limit the claimed subject matter to the precise formdisclosed. Many modifications and variations are possible in light ofthe above teaching. Further, any or all of the aforementioned alternateimplementations may be used in any combination desired to formadditional hybrid implementations of the Data Transfer Tool. It isintended that the scope of the Data Transfer Tool be limited not by thisdetailed description, but rather by the claims appended hereto. Althoughthe subject matter has been described in language specific to structuralfeatures and/or methodological acts, it is to be understood that thesubject matter defined in the appended claims is not necessarily limitedto the specific features or acts described above. Rather, the specificfeatures and acts described above are disclosed as example forms ofimplementing the claims and other equivalent features and acts areintended to be within the scope of the claims.

What has been described above includes example implementations. It is,of course, not possible to describe every conceivable combination ofcomponents or methodologies for purposes of describing the claimedsubject matter, but one of ordinary skill in the art may recognize thatmany further combinations and permutations are possible. Accordingly,the claimed subject matter is intended to embrace all such alterations,modifications, and variations that fall within the spirit and scope ofdetailed description of the Data Transfer Tool described above.

In regard to the various functions performed by the above describedcomponents, devices, circuits, systems and the like, the terms(including a reference to a “means”) used to describe such componentsare intended to correspond, unless otherwise indicated, to any componentwhich performs the specified function of the described component (e.g.,a functional equivalent), even though not structurally equivalent to thedisclosed structure, which performs the function in the hereinillustrated exemplary aspects of the claimed subject matter. In thisregard, it will also be recognized that the foregoing implementationsinclude a system as well as a computer-readable storage media havingcomputer-executable instructions for performing the acts and/or eventsof the various methods of the claimed subject matter.

There are multiple ways of realizing the foregoing implementations (suchas an appropriate application programming interface (API), tool kit,driver code, operating system, control, standalone or downloadablesoftware object, or the like), which enable applications and services touse the implementations described herein. The claimed subject mattercontemplates this use from the standpoint of an API (or other softwareobject), as well as from the standpoint of a software or hardware objectthat operates according to the implementations set forth herein. Thus,various implementations described herein may have aspects that arewholly in hardware, or partly in hardware and partly in software, orwholly in software.

The aforementioned systems have been described with respect tointeraction between several components. It will be appreciated that suchsystems and components can include those components or specifiedsub-components, some of the specified components or sub-components,and/or additional components, and according to various permutations andcombinations of the foregoing. Sub-components can also be implemented ascomponents communicatively coupled to other components rather thanincluded within parent components (e.g., hierarchical components).

Additionally, one or more components may be combined into a singlecomponent providing aggregate functionality or divided into severalseparate sub-components, and any one or more middle layers, such as amanagement layer, may be provided to communicatively couple to suchsub-components in order to provide integrated functionality. Anycomponents described herein may also interact with one or more othercomponents not specifically described herein but generally known toenable such interactions.

What is claimed is:
 1. A computer-implemented process, comprising:applying an application programming interface (API) to extract aplurality of data items from text being rendered on a display device inresponse to user interaction with a source application executing on acomputing device; applying a predefined application content modelcorresponding to the source application to identify and label theextracted data items; applying a subscription process to authorize andsubscribe one or more destination applications to access the identifiedand labeled extracted data items; and automatically applying one or moreof the identified and labeled extracted data items to populate one ormore corresponding data fields of one or more of the authorizeddestination applications.
 2. The computer-implemented process of claim 1further comprising: perform an automated search to retrieve informationrelevant to one or more of the extracted data items; and present anotification in an assistant app that is unrelated to either the sourceapplication or the destination applications to present the retrievedinformation relevant to one or more of the extracted data items.
 3. Thecomputer-implemented process of claim 2 further comprising presentingthe notification in the assistant app as an overlay on the sourceapplication.
 4. The computer-implemented process of claim 2 furthercomprising presenting the notification in the assistant app as anoverlay on one or more of the destination applications.
 5. Thecomputer-implemented process of claim 2 further comprising presentingsponsored content in the notification.
 6. The computer-implementedprocess of claim 1 further comprising automatically updating theapplication content model when an analysis of the source applicationindicates changes to the source application binaries.
 7. Thecomputer-implemented process of claim 1 further comprising: modelinguser behaviors with respect to the source application; and automaticallyinitiating one or more destination applications in response to themodeled user behaviors and one or more of the extracted data items. 8.The computer-implemented process of claim 1 further comprising: modelinguser behaviors with respect to one or more of the source application andthe destination applications; and applying the modeled user behaviors incombination with and one or more of the extracted data items tocustomize a digital assistant.
 9. The computer-implemented process ofclaim 1 further comprising providing at least a portion of theidentified and labeled extracted data items to one or more destinationapps running on one or more different computing devices of the user. 10.The computer-implemented process of claim 1 further comprising providingat least a portion of the identified and labeled extracted data items toone or more destination apps running on one or more of differentcomputing devices of different users.
 11. The computer-implementedprocess of claim 1 further comprising providing anonymized versions ofat least a portion of the identified and labeled extracted data items toone or more computing devices of other users.
 12. A computer-readablestorage device having computer executable instructions stored therein,said instructions causing a computing device to execute a methodcomprising: applying an application programming interface (API) toextract a plurality actions and corresponding attributes from contentand user interface (UI) controls being rendered in a presentation layerof a source app; applying a content model of the source app to formatthe extracted actions and corresponding attributes as app-independenttasks; and applying one or more of the app-independent tasks to populateone or more variables of one or more of destination apps that aresubscribed to receive those app-independent tasks.
 13. Thecomputer-readable storage device of claim 12 further comprising apresenting a user interface for setting an authorization state forsubscriptions to receive app-independent tasks.
 14. Thecomputer-readable storage device of claim 12 further comprising:performing an automated search to retrieve information relevant to oneor more of the app-independent tasks; and applying an assistant app topresent the retrieved information as a user selectable popup on thesource application.
 15. The computer-readable storage device of claim 12further comprising automatically updating the content model when ananalysis of the source app indicates that the source app has beenupdated.
 16. The computer-readable storage device of claim 12 furthercomprising: modeling user behaviors with respect to the source app; andautomatically initiating one or more destination apps in response to themodeled user behaviors and one or more of the app-independent tasks. 17.The computer-readable storage device of claim 12 further comprisingproviding at least a portion of the app-independent tasks to one or moredestination apps running on one or more different computing devices ofthe user.
 18. A system, comprising: a general purpose computing device;and a computer program comprising program modules executable by thecomputing device, wherein the computing device is directed by theprogram modules of the computer program to: scrape user-enteredinformation from a presentation layer of an arbitrary application;perform an automated search to retrieve information relevant to thescraped information; and present a notification in an assistant app thatis unrelated to the arbitrary app to present the information relevant tothe scraped information.
 19. The system of claim 18 wherein theassistant app is a digital assistant and further comprising presentingthe notification via a speech output of the digital assistant.
 20. Thesystem of claim 18 further comprising a program module for providing thescraped information to one or more third-party apps that have priorauthorization to receive that scraped information.